Computer vision based classification of fruits and vegetables for self-checkout at supermarkets

The field of machine learning, and, in particular, methods to improve the capability of machines to perform a wider variety of generalised tasks are among the most rapidly growing research areas in today’s world. The current applications of machine learning and artificial intelligence can be divided into many significant fields namely computer vision, data sciences, real time analytics and Natural Language Processing (NLP). All these applications are being used to help computer based systems to operate more usefully in everyday contexts. Computer vision research is currently active in a wide range of areas such as the development of autonomous vehicles, object recognition, Content Based Image Retrieval (CBIR), image segmentation and terrestrial analysis from space (i.e. crop estimation). Despite significant prior research, the area of object recognition still has many topics to be explored. This PhD thesis focuses on using advanced machine learning approaches to enable the automated recognition of fresh produce (i.e. fruits and vegetables) at supermarket self-checkouts. This type of complex classification task is one of the most recently emerging applications of advanced computer vision approaches and is a productive research topic in this field due to the limited means of representing the features and machine learning techniques for classification. Fruits and vegetables offer significant inter and intra class variance in weight, shape, size, colour and texture which makes the classification challenging. The applications of effective fruit and vegetable classification have significant importance in daily life e.g. crop estimation, fruit classification, robotic harvesting, fruit quality assessment, etc. One potential application for this fruit and vegetable classification capability is for supermarket self-checkouts. Increasingly, supermarkets are introducing self-checkouts in stores to make the checkout process easier and faster. However, there are a number of challenges with this as all goods cannot readily be sold with packaging and barcodes, for instance loose fresh items (e.g. fruits and vegetables). Adding barcodes to these types of items individually is impractical and pre-packaging limits the freedom of choice when selecting fruits and vegetables and creates additional waste, hence reducing customer satisfaction. The current situation, which relies on customers correctly identifying produce themselves leaves open the potential for incorrect billing either due to inadvertent error, or due to intentional fraudulent misclassification resulting in financial losses for the store. To address this identified problem, the main goals of this PhD work are: (a) exploring the types of visual and non-visual sensors that could be incorporated into a self-checkout system for classification of fruits and vegetables, (b) determining a suitable feature representation method for fresh produce items available at supermarkets, (c) identifying optimal machine learning techniques for classification within this context and (d) evaluating our work relative to the state-of-the-art object classification results presented in the literature. An in-depth analysis of related computer vision literature and techniques is performed to identify and implement the possible solutions. A progressive process distribution approach is used for this project where the task of computer vision based fruit and vegetables classification is divided into pre-processing and classification techniques. Different classification techniques have been implemented and evaluated as possible solution for this problem. Both visual and non-visual features of fruit and vegetables are exploited to perform the classification. Novel classification techniques have been carefully developed to deal with the complex and highly variant physical features of fruit and vegetables while taking advantages of both visual and non-visual features. The capability of classification techniques is tested in individual and ensemble manner to achieved the higher effectiveness. Significant results have been obtained where it can be concluded that the fruit and vegetables classification is complex task with many challenges involved. It is also observed that a larger dataset can better comprehend the complex variant features of fruit and vegetables. Complex multidimensional features can be extracted from the larger datasets to generalise on higher number of classes. However, development of a larger multiclass dataset is an expensive and time consuming process. The effectiveness of classification techniques can be significantly improved by subtracting the background occlusions and complexities. It is also worth mentioning that ensemble of simple and less complicated classification techniques can achieve effective results even if applied to less number of features for smaller number of classes. The combination of visual and nonvisual features can reduce the struggle of a classification technique to deal with higher number of classes with similar physical features. Classification of fruit and vegetables with similar physical features (i.e. colour and texture) needs careful estimation and hyper-dimensional embedding of visual features. Implementing rigorous classification penalties as loss function can achieve this goal at the cost of time and computational requirements. There is a significant need to develop larger datasets for different fruit and vegetables related computer vision applications. Considering more sophisticated loss function penalties and discriminative hyper-dimensional features embedding techniques can significantly improve the effectiveness of the classification techniques for the fruit and vegetables applications

Texture-based latent space disentanglement for enhancement of a training dataset for ANN-based classification of fruit and vegetables

The capability of Convolutional Neural Networks (CNNs) for sparse representation has significant application to complex tasks like Representation Learning (RL). However, labelled datasets of sufficient size for learning this representation are not easily obtainable. The unsupervised learning capability of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) provide a promising solution to this issue through their capacity to learn representations for novel data samples and classification tasks. In this research, a texture-based latent space disentanglement technique is proposed to enhance learning of representations for novel data samples. A comparison is performed among different VAEs and GANs with the proposed approach for synthesis of new data samples. Two different VAE architectures are considered, a single layer dense VAE and a convolution based VAE, to compare the effectiveness of different architectures for learning of the representations. The GANs are selected based on the distance metric for disjoint distribution divergence estimation of complex representation learning tasks. The proposed texture-based disentanglement has been shown to provide a significant improvement for disentangling the process of representation learning by conditioning the random noise and synthesising texture rich images of fruit and vegetables

Class distribution-aware adaptive margins and cluster embedding for classification of fruit and vegetables at supermarket self-checkouts

The complex task of vision based fruit and vegetables classification at a supermarket self-checkout poses significant challenges. These challenges include the highly variable physical features of fruit and vegetables i.e. colour, texture shape and size which are dependent upon ripeness and storage conditions in a supermarket as well as general product variation. Supermarket environments are also significantly variable with respect to lighting conditions. Attempting to build an exhaustive dataset to capture all these variations, for example a dataset of a fruit consisting of all possible colour variations, is nearly impossible. Moreover, some fruit and vegetable classes have significant similar physical features e.g. the colour and texture of cabbage and lettuce. Current state-of-the-art classification techniques such as those based on Deep Convolutional Neural Networks (DCNNs) are highly prone to errors resulting from the inter-class similarities and intra-class variations of fruit and vegetable images. The deep features of highly variable classes can invade the features of neighbouring similar classes in a learned feature space of the DCNN, resulting in confused classification hyper-planes. To overcome these limitations of current classification techniques we have proposed a class distribution-aware adaptive margins approach with cluster embedding for classification of fruit and vegetables. We have tested the proposed technique for cluster-based feature embedding and classification effectiveness. It is observed that introduction of adaptive classification margins proportional to the class distribution can achieve significant improvements in clustering and classification effectiveness. The proposed technique is tested for both clustering and classification, and promising results have been obtained

A sample weight and adaboost CNN-based coarse to fine classification of fruit and vegetables at a supermarket self-checkout

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. The physical features of fruit and vegetables make the task of vision-based classification of fruit and vegetables challenging. The classification of fruit and vegetables at a supermarket self-checkout poses even more challenges due to variable lighting conditions and human factors arising from customer interactions with the system along with the challenges associated with the colour, texture, shape, and size of a fruit or vegetable. Considering this complex application, we have proposed a progressive coarse to fine classification technique to classify fruit and vegetables at supermarket checkouts. The image and weight of fruit and vegetables have been obtained using a prototype designed to simulate the supermarket environment, including the lighting conditions. The weight information is used to change the coarse classification of 15 classes down to three, which are further used in AdaBoost-based Convolutional Neural Network (CNN) optimisation for fine classification. The training samples for each coarse class are weighted based on AdaBoost optimisation, which are updated on each iteration of a training phase. Multi-class likelihood distribution obtained by the fine classification stage is used to estimate a final classification with a softmax classifier. GoogleNet, MobileNet, and a custom CNN have been used for AdaBoost optimisation, with promising classification results

A comprehensive review of fruit and vegetable classification techniques

Recent advancements in computer vision have enabled wide-ranging applications in every field of life. One such application area is fresh produce classification, but the classification of fruit and vegetable has proven to be a complex problem and needs to be further developed. Fruit and vegetable classification presents significant challenges due to interclass similarities and irregular intraclass characteristics. Selection of appropriate data acquisition sensors and feature representation approach is also crucial due to the huge diversity of the field. Fruit and vegetable classification methods have been developed for quality assessment and robotic harvesting but the current state-of-the-art has been developed for limited classes and small datasets. The problem is of a multi-dimensional nature and offers significantly hyperdimensional features, which is one of the major challenges with current machine learning approaches. Substantial research has been conducted for the design and analysis of classifiers for hyperdimensional features which require significant computational power to optimise with such features. In recent years numerous machine learning techniques for example, Support Vector Machine (SVM), K-Nearest Neighbour (KNN), Decision Trees, Artificial Neural Networks (ANN) and Convolutional Neural Networks (CNN) have been exploited with many different feature description methods for fruit and vegetable classification in many real-life applications. This paper presents a critical comparison of different state-of-the-art computer vision methods proposed by researchers for classifying fruit and vegetable

Determining the factors associated with Unmet need for family planning: A cross-sectional survey in 49 districts of Pakistan

Introduction & Background: Around 137 million women in the developing world who would like to avoid childbearing are unable to do so, despite a huge increase in contraceptive access and use globally. Ironically, the prevalence of unmet need in Pakistan is among the highest in the world despite being one of the first countries in South Asia to launch national family planning program. The aim of this paper is to estimate the prevalence of unmet need for contraception and to indentify the factors associated with it.Methods: A cross-sectional survey was conducted in forty nine districts of Pakistan across all four provinces from September 2008 to March 2009. Using an adapted version of PDHS questionnaire, interviews were conducted with approximately 10,000 married women of reproductive age in each district. Sample was later weighted according to district population at the time of analysis to control over and under representation. Logistic regression analysis was used to assess the association between risk factors and unmet need.Results: The total unmet need for contraception was 23.5%. Multivariable analysis showed that unmet need was found significantly higher in Balochistan and Sindh province compare to Punjab. The unmet need was quite prevalent among the specific groups that include older age women, low or uneducated women, those who have higher number of living children, had no history of miscarriage or abortion, those who are not exposed to mass media once a week, and among the women in lowest wealth quintiles.Conclusion: Despite all the efforts made to increase in uptake of contraceptive method the contraceptive prevalence rate has hardly changed over the last decade. However, several groups of women continue to have high unmet need for family planning. Thus, the family planning programmes may need to shift their focus from increasing uptake of contraceptives to satisfying unmet need for contraception with special focus on those underserved marginalized groups and areas with highest levels of unmet need

The logistics of voucher management: The underreported component in family planning voucher discussions

Background: The purpose of health care vouchers or coupons is to receive a health service in exchange which is fully or partially subsidized, such as any treatment offered for communicable disease; for immunization; antenatal care-/postnatal care-related maternal health services; a family planning (FP) service; or to get a health commodity like a medicine. Vouchers are targeted for a group of people who can benefit the most such as on the basis of poverty ranking, marginalized or living in rural areas. According to the World Health Organization, voucher schemes in the area of sexual and reproductive health are considered of high value if they are implemented to address the issues of contraceptive commodity or service unavailability or to address the barriers to access such services through contracting out health services, for example, through social franchising (SF). FP vouchers can substantially expand contraceptive access and choice and empower the underserved populations. Literature cites voucher\u27s effectiveness in better targeting, increasing use, and improving program outcomes in FP programs; however, there is little research or explanation of how voucher management is done in practice.Discussion: The paper attempts to describe various components of voucher management system and its functioning using example of a voucher program in Pakistan. There are challenges such as high upfront cost, targeting the appropriate clients, validation of vouchers, and quality assurance, but these can be managed with better preparation at the planning and design stage. Strong monitoring and evaluation are integral to successful implementation of the voucher program. Also, voucher interventions that are targeted and adopt a pro-poor strategy have been found to improve access to care within poor and marginalized populations. Such programs have the capacity to bridge health inequities in developing nations. Targeted voucher schemes such as those which are designed as pro-poor or pro-rural are known to reduce barriers to access for those living with poverty or for the ones considered as marginalized population. Hence, such interventions have the capacity to fulfill the gaps in health inequities, especially, in low- and/or middle-income countries.Conclusion: Voucher programs should report the voucher logistics and management to build a larger evidence base of best practices. All voucher schemes must be designed, implemented, and evaluated on the basis of set objectives through addressing the local context. But any voucher implementing organization also conducting the in-house voucher management simultaneously may be considered as a weakness in program design, in turn providing rationale for either failure or success of that particular voucher intervention. Therefore, separating implementation and management of a voucher initiative can lead to enhanced transparency, improved accountability, allow for independent validation of services, and facilitate compliance for payments

A study protocol : using demand-side financing to meet the birth spacing needs of the underserved in Punjab Province in Pakistan

Background: High fertility rates, unwanted pregnancies, low modern contraceptive prevalence and a huge unmet need for contraception adversely affect women's health in Pakistan and this problem is compounded by limited access to reliable information and quality services regarding birth spacing especially in rural and underserved areas. This paper presents a study protocol that describes an evaluation of a demand-side financing (DSF) voucher approach which aims to increase the uptake of modern contraception among women of the lowest two wealth quintiles in Punjab Province, Pakistan. Methods/Design: This study will use quasi-experimental design with control arm and be implemented in: six government clinics from the Population Welfare Department; 24 social franchise facilities branded as `Suraj' (Sun), led by Marie Stopes Society (a local non-governmental organization); and 12 private sector clinics in Chakwal, Mianwali and Bhakkar districts. The study respondents will be interviewed at baseline and endline subject to voluntary acceptance and medical eligibility. In addition, health service data will record each client visit during the study period. Discussion: The study will examine the impact of vouchers in terms of increasing the uptake of modern contraception by engaging private and public sector service providers (mid-level and medical doctors). If found effective, this approach can be a viable solution to satisfying the current demand and meeting the unmet need for contraception, particularly among the poorest socio-economic group

Post-abortion care family planning use in Pakistan

Introduction: The stagnated CPR and high unmet need for contraception lead to approximately 890,000 induced abortions every year in Pakistan. A fairly recent study from Pakistan also revealed that around 40% of abortions are performed by unskilled workers in backstreet clinics. Considering these grave statistics, it should not come as surprise that unwanted pregnancies are the leading cause of induced abortions in Pakistan. Despite country\u27s inferior situation, there is no data available in Pakistan that unveils the much needed information pertaining to post-abortion care family planning (PAC) use. Thus, this paper attempts to document socio-demographic profile seeking post-abortion care clients; estimate proportion of post-abortion contraception uptake and determine its associated factors.Methods: Medical records of 17,262 women seeking PAC as a result of incomplete abortion and treatment for complications arising from unsafe abortions were analyzed. The associations between risk factors and post-abortion family planning uptake were assessed by applying univariate and multivariable logistic regression.Results: High post abortion contraceptive use (72.9%) was observed amongst the women who had sought for PAC services. where, 66% of the women opted to use short-term methods. The rest (33.5) considered long-term reversible IUD and implant as their method of choice and only 0.4% had undergone voluntary sterilization. Multiple logistic model identified province, women education, women occupation status, monthly family income, first time visitors to the centre, previous contraceptive use, and type of PAC treatment provided, women\u27s health condition after post-abortion treatment had significant associations with the uptake of contraception.Conclusion: The present study highlights the importance of strengthening post-abortion family planning services in the country which will not only contribute in increasing the overall contraceptive use in the country but will also prevent high unintended pregnancies that may ultimately lead to induced abortions

IUD discontinuation rates, switching behavior, and user satisfaction: Findings from a retrospective analysis of a mobile outreach service program in Pakistan

Background: In Pakistan, the uptake rate for the intrauterine device (IUD) is very low at 2.5%. The most popular modern contraceptive methods in Pakistan are female sterilization and use of condoms. The Marie Stopes Society established its mobile outreach service delivery program with the aim of increasing use of modern quality contraceptive services, including the long-term reversible IUD, by women living in hard-to-reach areas. The present study attempts to assess IUD discontinuation rates and associated factors, including switching behavior and level of satisfaction with this type of service delivery.Methods: Using a cross-sectional approach, we contacted 681 women who had received an IUD from the Marie Stopes Society mobile outreach program during July and August 2009. Successful interviews were conducted with 639 of these women using a structured questionnaire. The data were analyzed with Stata 11.2 using simple descriptive Chi-square and Cox proportional techniques.Results: Analysis revealed that 19.4% (95% confidence interval 16.3-22.5) of the women discontinued use of their IUD at 10 months and, of these women, the majority (69.4%) cited side effects as the main reason for discontinuation. Other factors, such as geographical catchment province, age of the woman, history of contraceptive use before IUD insertion, and side effects following insertion of the device, were found to be significantly associated with IUD. Amongst the women who had their IUD removed, 56.5% did not switch to any other contraceptive method, while 36.3% switched to either short-term or traditional methods, such as withdrawal, rhythm, and folk methods. Degree of satisfaction with the device was also significantly associated with discontinuation.Conclusion: Early discontinuation and not switching to another contraceptive method increases the risk of unplanned pregnancy. Health care workers should be trained in managing clients\u27 concerns about the IUD to prevent discontinuation and providing counseling services for clients to select an alternative contraceptive method if they decide to discontinue